Making a virtual memory manager for 4KB pages, adding a PF "handler", and enabling paging.This switches from only using physical RAM to using virtual memory & pages, allowing "any" memory address (assuming it falls within the 4GB limit for this 32bit OS) if it's mapped to a valid physical address. E.g. setting the entry point for programs to 0x400000/4MB, or using the same addresses in multiple processes.
Also showing how to boot on hardware, and it works! But only on 1 old laptop so far.

Part 2 will be malloc/free C functions, system calls, and test program. This was split into 2 parts to have slightly less run time than an extended LoTR movie.

OS Dev playlist: https://www.youtube.com/playlist?list=PLT7NbkyNWaqajsw8Xh7SP9KJwjfpP8TNX

Source code for this video & upcoming videos:
https://git.sr.ht/~queso_fuego/amateur_os_video_sources/tree/master/item/42_vmem_paging_malloc_free_high_half_kernel

Git Repos:
Sourcehut: https://git.sr.ht/~queso_fuego/quesos
Github mirror: https://github.com/queso-fuego/amateuros

Links to things:
DD for windows: http://www.chrysocome.net/dd
Virtual memory/paging tutorial: http://www.brokenthorn.com/Resources/OSDev18.html
https://wiki.osdev.org/Paging
https://wiki.osdev.org/TLB
https://wiki.osdev.org/Setting_Up_Paging
https://wiki.osdev.org/Identity_Paging
https://wiki.osdev.org/Higher_Half_Kernel

Join the Community Discord:
https://discord.gg/yKm4T89QFn

Contact:
email - fuegoqueso@gmail.com
twitter - @Queso_Fuego
twitch - https://www.twitch.tv/queso_fuego

Notes:
- The brokenthorn page used 0x7FFFF000 for page table/dir frames, but I'm not sure why. Bits 12-31 is 20 bits, up to 0xFFFFF000. 19 bits is up to 0x7FFFF000, so I think maybe they miscalculated, and I blindly copied :)
I can update to use the full 20 bit range 0xFFFFF000 in the future.

- I don't like some things in the virtual memory manager. Using GET/SET_FRAME naming or plain bitwise operations to set/clear flags may be clearer and simpler. If typedef-ed arrays worked differently I'd use page_table[i] instead of pt with entries[i], but a typedef means the type is actually (*)[1024] and needs a type cast, which is annoying. 
Not using typedef-ed arrays is simpler but then you don't have page_table* or other specific named types. And I could set entries more succinctly with "page = ADDRESS | FLAGS" instead of more lines for SET_ATTRIBUTE and SET_FRAME.
table and table3G could be switched to make more sense, it looks wrong right now.
Always easy to see refactors after the video is done :)

- The kernel isn't really in higher half memory yet. It's mapped to 0xC0000000+, but if 3GB+ addresses aren't used, and they aren't currently, the kernel is still in identity mapped addresses 0x0-0x400000. I need to Link the kernel to 0xC0000000+, Load kernel code to higher half addresses, and Jump to 0xC0000000 to execute the kernel from the higher addresses.
This means doing memory/paging setup before the kernel is jumped to. I made a C file for this, that the bootloader jumps to, that sets up pre-kernel memory and paging, loads the kernel to higher half addresses, removes the lower half mapping, and calls the kernel. 
That should be on the video after the next one.

- Commands to show the current physical memory (what files/programs are in memory at which blocks) and virtual memory (what files/programs are at which virtual & physical addresses) would be useful, at least for debugging.

The next video will be malloc/free C functions & syscalls, then an actual higher half kernel at 0xC0000000, not lower half memory. After that:
- A better filesystem incl. LBA & directories
- Editor updates: scrolling, line numbers, etc.
- Processes/multitasking
- Starting an assembler
or other things...

- Let me know if there's anything specific you'd like to see!

Outline:
0:00 Intro/how I boot on hardware (thinkpad x60)
15:58 Paging overview/motivation
37:48 virtual_memory_manager.h
1:57:40 Page fault handler, enable paging in kernel
2:18:25 Fix check_filename(), finding free blocks
2:31:06 Remove hardcoded filenames in makefile
2:37:26 Change loading files in kernel to use paging
3:00:35 Outro/Coming up

Software used:
VMware Workstation Player: https://www.vmware.com/products/workstation-player/workstation-player-evaluation.html
freeBSD: https://www.freebsd.org/
qemu: https://www.qemu.org/
vim: https://www.vim.org/
clang: https://clang.llvm.org/
davinci resolve: https://www.blackmagicdesign.com/products/davinciresolve/
OBS Studio: https://obsproject.com/

Peripherals:
Camera: Sony ZV-1, Elgato Camlink 4k
Microphone: Shure SM7B, Cloudlifter CL-1, Focusrite Scarlett Solo
Keyboard: HHKB Pro Hybrid Type S, white/blank keycaps
Mouse: Logitech M590

Updates may be on twitter, YouTube channel community tab, community discord, or mailing lists. If I remember.

Credits:
Music:
Acid Trumpet Kevin MacLeod (incompetech.com)
Licensed under Creative Commons: By Attribution 3.0 License http://creativecommons.org/licenses/by/3.0/

#osdev #memory #paging